Today's computer networks are extremely complex, with hundreds or more of applications, thousands or more of servers, hundreds or more of locations, hundreds of thousands of clients, and network traffic routed by numerous switches and routers on the computer networks. Network and application data collected from various parts of the network can provide insight into network conditions, but the enormous amount of data present a challenge for data storage, processing, and retrieval.
Many conventional network monitoring systems store data packets in storage devices on a first-in-first-out (FIFO) basis. These network monitoring systems store the data packets in the storage devices upon receipt. When operations such as analyzing network conditions or performance are called for, the data packets stored in the storage devices are retrieved and then analyzed. However, the network monitoring systems have limited storage capacity. Hence, the network monitoring systems can store data packets captured within a limited time frame. As data rate increases, the network monitoring systems can store data packets spanning over a shorter amount of time. The shorter retention of data packets may result in incomplete or inaccurate analysis. To store data packets for a longer time, the network monitoring systems must be equipped with more storage resources.
Moreover, various operations may involve extensive manual or automatic processing on the data packets. These operations may involve retrieval and analysis of a large number of data packets, which may take an extensive amount of time before the results of the operations are available. The extensive amount of time for retrieving and processing the data packets may result in belated responses or lost opportunity to detect and resolve issues. To obtain faster results of analysis, the network monitoring systems must be equipped with additional computation resources.